Monster Media 1996 #15/Monster Media Number 15 (Monster Media)(July 1996).ISO/bbs_util/wg960530.zip/CHANGES.DOC

Text File | 1996-05-30 | 28KB | 678 lines

     W...W
     W.W.W
     .W.W.

     WaterGate
     Mailprocessor for Fidonet & Usenet
     Changes and fixes to the previous version

     (c) Copyright 1993-1996 WaterLine Software Development
                             All Rights Reserved

May 30th, 1996

Hello there!

It has been quite some time since the release of 0.90, but there
have been quite some things going one. No more but 12 different
patch files were release between these two major releases to test
new functionality and solve bugs.

I attempted to make the gateway completely transparent by putting
the Fido MSGID kludges in the Internet / Usenet Message-ID header
and the other way around and extracting and restoring them on the
way back. This way a proper In-reply-to header could be constructed
from the REPLY kludge, which was the original MSGID created on the
way in. This unfortunately didn't work.

A lot of tossers (including WaterGate in the beginning, <ahum>)
didn't like the new format of the MSGID kludge and were expecting
a valid Fido address as the first argument. I had to take out that
code again. Good news is that the code for putting Fidonet MSGID
and REPLY kludges in Internet / Usenet message headers is still
there though, so that part becomes transparent and allows linking
of replies, for one.

All these changes, testing, patches, awaiting feedback, fixing, etc.
took a lot of time, but the end result is that 0.91 is a stable
improvement over 0.90. See the new future.doc file for changes you
can expect in the near future.

Back to business..

Ramon


This file describes the changes and fixes that you will find in
version 0.91 of the WaterGate package. This version contains al lot
of bug fixes, partitial rewrites and some new features as listed
below.

 ■ Memory requirements have gone down drastically. WaterGate will
   now used almost all the available "low" memory and won't write
   messages to the "too big" directory as soon as it hits the 70k
   limit anymore.

 ■ Points of nodes with a domain address are now automatically
   mapping to username@p<point>.<user's domain address>. For
   example, point 16 of node 2:200/111 with domain address
   cold.wline.se will now get the e-mail address
   [e-mail] and incoming mail for this address is
   automatically mapped to the correct AKA.

 ■ Squish base code and the WtrUtil program are now a lot more
   stable, have stopped destroying message bases all the time.

 ■ MSGID and REPLY kludges are now encoded in the Message-ID and
   In-Reply-To: headers and decoded to the original format on the
   way back.

 ■ Encoded files are now automatically decoded and stored in a
   special directory. Currently only when importing a message in
   a *.MSG base, but more functionality will follow as soon as
   some problems have been solved (split part counting, for one).

 ■ Support for 25+ lines screens (windows) has been improved.
   Centered menus, longer listings, not restoring to 25 lines
   on exit, etc.


Following is the complete list of bugfixes and other improvements
between versions 0.90 and 0.91, in reverse order of fixing date.
PLEASE READ IT carefully (I know it is a lot) and see if something
relates to you. If it does, please check that it is still
functioning!

 ■ Now allowing continuation lines with MIME parameters when decoding.
   Especially the NAME one.

 ■ Fixed some cleanup after Squish Purge, Link and re-index.

 ■ Removed the FD_Using log action.

 ■ Netmail on Hold is now processed when not running in FrontDoor mode.

 ■ Centered the copyright/beta info and about boxes.

 ■ Allowed quite some lists to use the full screen length instead of a
   fixed length (groups list, areas list, users list, etc.)

 ■ Fixed some problems with writing the progress outside of the centered
   percentage done windows on 25+ lines windows.

 ■ Video mode is no longer changed on exit if it wasn't set at startup.
   This stop the 43 lines window from resetting to 25 lines. Neither
   when crashing.

 ■ Fixed two potential memory freeing errors found via the P10 crashes.

 ■ Mail -> Netmail translation now uses the Gateway AKA in the MSGID
   instead of the first system AKA.

 ■ Bad-case Message-ID -> MSGID conversion now builds the default MSGID
   string instead of returning "0 00000000".

 ■ Fixed a small bug in the uu-decoding to do with the last line, but
   only when it didn't contain a three-pair in the end.

 ■ Generated e-mail address for fido user will now contain "p0" when
   the gateway aka is a point address as well. Without this information
   it is impossible to map back to the node number (without the point).
   Added for both normal and small address generation.

 ■ Small modifications the command-line help screens for WtrUtil and
   WtrConf.

 ■ Took out some old UUCP routing code data structures and memory usage.

 ■ WtrUtil purge no longer calls the purging code when age and limit
   are both set to 0.

 ■ WtrUtil Squish Purge code now closes the message base when done.

 ■ Attempted to fix the code that purges Squish and JAM areas by age
   to prevent calculation errors on the maximum age of messages.

 ■ Removed "Not my outbound" in FrontDoor netmail checking code.

 ■ Added logging for news articles that are not accepted. The names of
   all the newsgroups it was posted in are logged.

 ■ Bounce reason is no longer up in all uppercase, but the case from
   the route.tdb file is kept.

 ■ Message-ID is no longer put into the MSGID header as this confuses
   editors and tossers and causes loads of problems.

 ■ Message-ID in new style MSGID was interpreted as a Fido address to
   form the complete From: aka, but completely failed. Built in some
   extra checks to prevent this from happening.

 ■ Fixed a bug in the encoded file extracter for MIME encoded files. The
   last or two last bytes could get lost at times.

 ■ Added prevention for bounced mail coming back to the list server after
   a distribution by checking for "MAILER-DAEMON" and writing the message
   to the bad area for inspection.

 ■ JAM base linking on disk should now work. A bug kept it from creating
   a temporary file used to store the link information.

 ■ Squish index file is no longer opened and closed unnecessary if
   there is nothing to update.

 ■ Last used Squish and JAM bases are now closed when WtrUtil Import
   is done.

 ■ Squish base index is now updated on disk before the base is unlocked.
   This should some problems in multi-tasker environments.

 ■ Read-ahead cache is now freed in advance when importing *.MSG msgs
   into a JAM or Squish base.

 ■ When importing *.MSG messages into a JAM or Squish base, the
   Soft-CRs are now removed.

 ■ New language entry for 1012: Disconnected from mailing list @1@.

 ■ Squish now closes the message base properly if an error occurs
   during a scan.

 ■ Number of bytes exported from Squish base is now counted properly.

 ■ Scanned Squish messages are now properly flagged as Sent.

 ■ Added log file toggles plus help to WtrConf for the two new toggles.

 ■ Added NOLOCALFLAG option to ROUTE.TDB to remove the Loc flag before
   importing a netmail into a *.MSG message base.

 ■ Message-ID: / In-Reply-To: contents are now properly kept during
   transition in a FTN network and MSGID / REPLY are now properly kept
   during transition on the Internet. The latter is the more important
   though.

 ■ Built the MessageID -> MSGID coding and MSGID -> Message-ID decoding.
   (wgmid$)

 ■ Built the MSGID -> Message-ID coding and Message-ID -> MSGID decoding.
   (wgcid$)

 ■ Optional In-Reply-To: is now translated into a REPLY: kludge.

 ■ Message-ID: is now translated (coded, actually) into a MSGID: kludge.
   Notice that translated mails never had MSGID up to now (oops...).

 ■ MSGID to Message-ID: and REPLY to In-Reply-To: or References: now
   in place.

 ■ Started generic mapping between FTN MSGID: / REPLY: kludge and UUCP
   Message-ID: / In-Reply-To: / References: header.

 ■ Added shrinking of the statistics file to WtrUtil + command line
   option "SHRINKSTA", like SHRINKLOG.

 ■ Small cosmetic change in the keys description line (last line on the
   screen) when attempting to blink while under a GUI.

 ■ WtrGate now ignores file attaches with archives that are not in
   WaterGate's outbound (happens with two tossers, two outbounds and
   one *.MSG netmail directory and FrontDoor style mailer).

 ■ Updated help for Sysop Name field in user record.

 ■ Added a new toggle for logging all the outbound UUCP filenames that
   area created (jobs).

 ■ Added a new toggle for logging all echomail messages processed from
   a .PKT file: size of the message plus area in which it came.

 ■ Added logging of "secure" or "normal" when processing a .PKT, based
   on the existing and correctness of a password in the .PKT file.

 ■ Renumbering an empty *.MSG area will now set the LASTREAD file to 1.

 ■ Fixed a problem with the ^APATH: line splitting. It would start the new
   line with the last used net number, even when not required.
   For example: 280/200/111.

 ■ The proper format p#.f#.n#.z#.xxx is now used when Small addresses in
   the gateway settings is enabled. Used to be z#.n#.f#.p#.xxx.

 ■ Echomail is now gated to news if the only target is a BAG style
   system. Before it only triggered on UUCP style systems.

 ■ Swapfile is now closed and deleted when wtrgate.exe cannot start.

 ■ Built in crash prevention when running low on memory in allocating
   memory for the language structures, at startup. It will now write
   a error line and abort.

 ■ If your computer has no XMS and Cache TDB files is enabled, then
   wtrgate.exe will now disable caching at start-up and write a line
   to the logfile. Instead of crashing when trying to use the XMS
   code anyway...

 ■ For binkley outbound, each of the attached files (subject line)
   are now processed and written to the outbound flow file. Also,
   the complete path to the file is stored. The file is never
   marked for kill after transfer.

 ■ MIME decoder now ignores continued headers and other "Content-"
   lines before the start of the encoded block.

 ■ Modified decoder for MIME support. It now accepts any type of
   Content-type that starts with "application/" and contains a
   filename on the same line (name="ramon.zip").

 ■ Moved check for AreaFix before check for mailing list when
   processing a netmail.

 ■ MAP-AREA was resetting the sender's user record nummer to gain
   full distribution and all subsequent news was thus no longer
   "from" that user, causing WaterGate to send a copy back to that
   user. You uplink, for example... ;(  It now resets and then
   restores the user record number.

 ■ Changed FORCENOBITMASK in ROUTE.TDB to try to produce the "old"
   style spool directory filenames and .XQT / .CMD contents. If the
   item below gives problems for your site, then try FORCENOBITMASK
   and see what happens. Please report it to me if you have to use
   this!!

 ■ The UUCPname in the .XQT and .CMD files is now no longer limited
   to 6 or 7 characters, but the entire UUCPname is put in where
   required. I hope this finally solves the problems people were
   having with their provider. (Only for people with a UUCPname of
   6 or more characters).

 ■ Fixed a bug in the automatic UU-encoding routines that would loose
   one or two bytes at the end of the file. This was breaking ZIP
   files, for example... ;(

 ■ Empty news batches (after decompression) will now be logged and
   skipped, instead of some strange error log message.

 ■ Decoding on netmail is from now on only done of messages for a
   system AKA. This way mail to your users won't be touched.

 ■ Added simple reporting in logfile when message is written to the
   BAD area.

 ■ Fixed three situations in SquishSaveMessage that could cause memory
   leaks and leave the message base locked.

 ■ Reduced memory requirements for SquishSaveMessage so it runs better
   on low memory.

 ■ Improved checks for available memory in SquishSaveMessage, is it
   doesn't start to complain too fast.

 ■ If SquishSaveMessage would run out of memory, it forgot to unlock
   the message base. Fixed.

 ■ Basic UU-decoding seems to work now when importing in *.MSG !! ;)

 ■ Relaxed the memory limits a little bit more to reduce log lines like
   "Low on memory, skipping message".

 ■ Improved Private Mail setup editor by using the automatic fields.

 ■ Rewrite of *.MSG save message routine for checking for enclosed files.

 ■ Written core module for extracting files and not modifying the
   internal message format (it needs to be exported to other users that
   don't want decoding!).

 ■ Added decoding path for netmail and private mail and toggles to
   enable/disable them plus entries in the helpfile.

 ■ Key number is now put in the tear line, PID kludge and Received header.

 ■ Improved "emptying" of new areabase records.

 ■ Checked Area Record editting in WtrConf for new fields. Updated
   helpfile as well.

 ■ New upgrade program for areabase changes only.

 ■ Added "Decode files" and "Files path" to Area Record.

 ■ Started adding decoding routines to WaterGate.

 ■ Added preventions for empty header lines when finding the reply
   address. For example "Reply-To: ".

 ■ Small modification to the selection of the Origanization line when
   the organization line in the user record is empty. Updated help
   file for the Organization lines as well.

 ■ Fixed a problem with incoming .X file not being deleted for mail
   messages to multiple receipent.

 ■ Fixed a serious problem with incoming .X/.D files for mail not
   being deleted. This was caused by the rewrite for processing
   mail in .BAG files.

 ■ Added checks for Local_Echomail in mailing list message distribution
   after my first test with a locally linked area resulted in a complete
   mess.

 ■ Added function to edit log option "read-only denied posting".

 ■ WtrConf, usenet compression options: changes "zip" into "gzip".

 ■ WtrGate no longer starts when a problem occurs with the swapfile.
   This can be the inability to create the swapfile, or not enough
   free space on the swap device. You can use this as well for checking
   the initial disk space and not swapping if it is too low.

 ■ Added logging for the item below. Can be turned off with the group
   switch "Log applied mapping statements".

 ■ Added support for points of nodes defined in your user base on the
   part of UUCP -> Fido translation. If for example the e-mail address
   p34.wsd.wline.se didn't have a direct mapping and wsd.wline.se is
   defined in your userbase as 2:200/111, then this e-mail will end up
   as a netmail to 2:200/111.34. Works for points only.
   Make sure you turn OFF the "Allow sub-domains" for fido style users!!!

 ■ MAP-UUCP statements (except with -UF) are now checked for point e-mail
   address building. If you have defined a mapping for a certain node and
   without a user name and the address to map is for a point and there
   is not specific mapping for that point address, then the node address
   mapping is used. For example:
   MAP-UUCP wsd.wline.se 2:200/111
   would be used to create 2:200/111.15 -> p15.wsd.wline.se
   but only under the above restrictions.
   Normally you would do this by defining a domain address in a user
   record, so the biggest use of this is for your system domain
   addresses, although it only works for that node and its points,
   not for entire nets or zones.

 ■ Points of your own system with whatever AKA defined, not necessarily
   the gateway AKA, now get the e-mail address
   p<nr>.<first system domain address>.

 ■ Points of a node defined in your userbase now get the e-mail address
   p<nr>.<domain address as defined in the user record>. But only _if_
   there is a domain address defined there. For example: 2:200/111 is
   wsd.wline.se, then 2:200/111.15 becomes p15.wsd.wline.se.

 ■ Changed resetting of Flag bits when packing a message (into a .PKT)
   according to FTS-0001.015 specifications. It now resets all except
   for Private, Crash, File attach, Receipt request, Is return receipt,
   Audit trial requested (bits 0,1,4,10,12,13,14).
   There was no request for it, but I ran into it and it seemed right.

 ■ Adjusted the new memory allocation scheme again to prevent problems
   due to memory fragmentation.

 ■ Duplicate Organization lines should no longer show up.

 ■ It is now possible to override the Newsgroups:, Organization: and
   Subject: lines in the body of a netmail or echomail. These headers
   will replace to headers that would otherwise have been put in the
   mail or news message.

 ■ Long headers defined in the body of a netmail or echomail are now
   completely copied into the header and no longer cut off because of
   the re-paragraphing.

 ■ Fixed a problem with empty netmails and strange "n" line numbers in
   the statistics file.

 ■ Fixed a problem with the new memory management method. The trigger
   for swapping was set below the mimimum memory limit.

 ■ Outgoing UUCP .XQT files now have the first 7 (instead of 6)
   characters of the UUCPname in the F and I lines. This hopefully
   solves some problems with news that is trashed at ISPs.

 ■ The OrigNet/Node generation in the packed messages (inside .PKT
   files) is now set to your system AKA, instead of the AKA of the
   original message poster. This to overcome the Squish node level
   security problem.

 ■ Added NOTESTORIGADDR to disable to new packed message OrigNet/Node
   generation.

 ■ MAP-AREA now allows for an optional third argument which becomes
   the REPLYADDR kludge when the news message is translated into an
   echomail message and possible imported into a message base. This
   allows for a forced reply address for mailing lists. All other
   reply addresses go into REPLYALSO kludges automatically.

 ■ Invalid MAP-AREA lines are no longer processed.

 ■ REPLYALSO kludges are now added when translating UUCP -> Fido for
   the other possible reply addresses (Reply-To:, Sender:, From:)
   which is not yet in REPLYADDR.

 ■ The correct reply header is now put into the REPLYADDR kludge when
   translating mail->netmail or news->echomail.

 ■ AreaFix and newsfix now support an extra space between the areaname
   and the + or - sign. (+<space>alt.bbs.watergate).

 ■ Mail .DAT files are no longer kept open until another .DAT file has
   to be created for the same UUCP style user.

 ■ Fixed a bug in the routine that searches the To: line from the .D
   file for an e-mail address. It didn't correctly browse all the
   substrings.

 ■ Added detection of MAP-FIDO to decision of processing netmail from
   message base or not.

 ■ MAP-FIDO can now be applied to ALL netmails that are processed by
   WaterGate. This is very dangerous, but also very powerful!

 ■ Fixed a problem with processing mail which originated by the new
   processing method.

 ■ Fixed a problem with the mailing list when looking up the correct
   spelling of a mailing list name. It returns the wrong field from
   a variant record... ;(

 ■ List Server will now send a listing of available mailing lists
   when a somebody tries to connect to a non-existing mailing list.

 ■ Added language entry 104 for failure of automatically included
   uu-encoded files.

 ■ Added an extra empty line between each echo of a command that the
   list server will process.

 ■ List Server now replies that it will send help or a listing of
   available mailing lists in another message. Added language entries
   1040 and 1041.

 ■ Unique 8-character hexadecimal numbers are now always in lowercase.
   This is much better looking, especially in Message-ID: headers.

 ■ Re-added intelligence that adds area descriptions to %LIST and
   %QUERY listings.

 ■ Added log lines for UUCP/Fido forward requests.

 ■ Added test for recovering an area from passive mode when a user
   connects it again.

 ■ Added test for putting an area in passive mode when disconnecting
   an area and during %-ALL.

 ■ Added CheckIfAreaCanGoPassive and CheckIfAreaHasToGoNonePassive.

 ■ FIDOREQ.LST is now always created in WaterGate's system directory,
   just like UUCPREQ.LST always is.

 ■ UUCPREQ.LST now _always_ has a plus or minus sign in front of the area
   name when the area has to be (dis)connected.

 ■ Added a message about the maximum of 5 MAP-UUCP statements when not
   registered.

 ■ Changed message "Trashing oversized message" to "Out of memory..",
   which is the real reason.

 ■ Minimum available memory for starting a toss or scan is now 50k.
   This is _after_ all the configuration tables have been loaded.

 ■ Lowest memory limits for swapping and trashing have been reduced to
   respectively 10k and 5k of free continuous memory.

 ■ Added new memory allocation mechanism for temporary buffers, allowing
   them maximum available memory while not immediately disturbing the
   swapping or trashing limits.

 ■ Added internal ability to free the read-ahead cache before starting
   to process the message, but keeping the file open file post-updates.
   This frees to 10k read-ahead cache when scanning netmail so more
   memory is available when exporting the message.

 ■ Fixed a small visual problem when running low on memory and aborting
   before having displayed the progress window.

 ■ Small adjustment for global memory usage statistics when not tossing
   or scanning anything.

 ■ Adjusted memory usage (-MEMUSAGE) counting for new newsgroup names
   filter.

 ■ Changed detection whether user is allowed to post in an area
   (regarding read-only groups). Now, if the user is connected to
   the area but the user has NO access to any of the groups the
   area is in, then the user is allowed to post there. Otherwise,
   if the user is in one or more of the groups the area is in,
   then the read/write check is done with the read-only groups
   filter. Both for news and echomail. Thus:
   Area  User   ReadOnly  Result
   A     A      Not A     Write access
   A     A      A         NO write access
   A     Not A  Not A     Write access      ** NEW **
   A     Not A  A         Write access      ** NEW **

 ■ Added option to log all imported netmail, inclusive a toggle to
   switch it off.

 ■ "#! rmail ..." lines in .BAG files are now checked for a 10-digit
   number right after the rmail line. This number is a message length
   indication and will be removed. Actually, this number of bytes
   should be processed... just don't put any valid #! rnews lines
   in a mail message inside a .BAG file, oke?

 ■ BAG supplier now creates correct .X and .D files for e-mail by
   truncating the UUCPname at six characters and using the new
   spool filenames.

 ■ Added FORCENOBITMASK option for the route.tdb file to disable
   the new spool filename generation. This to keep all the batchfiles
   out there alive until they have been adjusted.

 ■ Files created in the spool directories now have the "munge bitmap
   character" added in front of them, which will always be 0 when
   created by WaterGate. This allows direct renaming to .X/.D for
   re-processing.

 ■ Added help screens for Allow Headers, Mail Grade, News Grade and
   Time Slicing.

 ■ Redid the System Settings and UUCP Settings. They now both use
   Automatic field definitions, so you can move from the top to
   the bottom item and back.

 ■ Added option to disable copying of headers from netmail/echomail to
   mail/news option: WtrConf, System Configuration, Gateway Settings,
   Allow headers.

 ■ Added Mail Grade and News Grade settings to WtrConf, System
   Configuration, UUCP settings.

 ■ Removed NOSLICE option for the ROUTE.TDB file.

 ■ Added time slicing setting to WtrConf, System Configuration, System
   settings.

 ■ Automatic uu-encoder now searches both inbound directories when
   there is no path specifier in the filename for the attached file.

 ■ Automatic uu-encoder no longer mentions a path of the attached
   file in the mail message.

 ■ "This message was rerouted at" is now inserted at the start of
   message body, instead of the start of the header, before all the
   kludges.

 ■ Spaces are now allowed between the AREA: kludge and the area name.

 ■ AREA: kludge in the middle of a netmail no longer changes it into
   an echomail message.

 ■ BAG supplier is now allowed to have e-mail inside the .BAG files.
   WaterGate writes these e-mails to .X and .D files in the BAG spool
   directory and processes those afterwards. This ie because of
   possible multiple receipients for the mail message. You can now
   see "Tossing for ..." for the BAG style user twice in the logfile:
   once for the .BAG files and once for the .X/.D files.

 ■ Added checks to prevent triggering on "#! rmail" in a news message
   or "#! news" in a mail message. BAG files accept both though and
   start a new message.

 ■ BAG user now searches for *.BAG instead of NEWS*.BAG.

 ■ AreaFix used language entry 2023 instead of 2024 when the user was
   already connected to the area.

 ■ List server and mailing lists now detect bounced messages from
   MAILER-DAEMON and then dump the message in the BAD area.

 ■ When scanning for file attaches, WaterGate now remembers all the
   archive names for each node. Condition is that they end with .SU#
   etc. Temporary debug logging to be sure.

 ■ File attach netmail with a Hold flag should now be scanned properly
   and the attach archived remembered and no new archive created for it.

 ■ Fixed an uninitialised boolean that caused the "Translating
   Echomail -> Netmail" unnecessary on ever news message. It thought
   it had to distribute the message via a mailing list.

 ■ Echomail that goes to the BAD area is now longer counted in the
   area's statistics.

 ■ Added support for checking existance of binkley outbound archive in
   other than main outbound directory, for example for other zones and
   for points. No longer will the archive extensions be used too much.

 ■ Fixed problem with "unable to complete parse of ..." because of
   extra spaces at the end of the line.

 ■ Fixed a PATH: line generator that didn't put the colon behind the
   kludge.


Please report bugs and problems to us at the addresses below. Please
complain about any little thing you find irritating. Two important
rules when reporting:

  1) PLEASE _ALWAYS_ MENTION THE VERSION YOU ARE USING!

  2) PLEASE ALWAYS MENTION THE MESSAGE BASE TYPE.
     When reporting a problem related to a message base.

Remember this program is in BETA phase and should be tested instead
of just being used. Without good beta testers, problems will never
be solved. We need feedback!!

Please read UPGRADE.DOC for upgrade details.

Good luck with the new version, and let us know!

Ramon

[e-mail]